{% load static %} Deep Learning: GANs {% include 'layout/components/css-head.html' %} {% include 'layout/components/navbar.html' %}

Generative Adversarial Networks and Expectation Maximization

Objectives:

  1. Implement a Generative Adversarial Network with the network structure proposed in Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
  2. Implement activation maximization, a visualization technique
In [1]:
# Import required libraries
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import math
import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'
%matplotlib inline

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

Some Background...

Generative vs Discriminative Models

Generative models try to model the distribution of data in an explicit way (by learning the joint probability distribution p(x,y)). This is in contrast to discriminative models that try to infer the output directly from the input (by learning the conditional probability p(y|x) or a mapping of the inputs x to outputs y). In other words, generative models describe how data is generated while disriminative models learn the boundaries to discriminate among classes.

GANs

GAN stands for Generative Adversarial Network. They are aptly named as such because GANs are essentially made up of two competing networks: a generator and a discriminator. The discriminator is a binary classifier with the two classes being "taken from the real data" ("real") and "generated by the generator" ("fake"). Its objective is to minimize the classification loss. The generator's objective is to generate samples so that the discriminator misclassifies them as real.

However, it is important to note that a good generator does not just learn to replicate the real data or find one perfect fake. The generator needs to be able to generate a variety of fake samples such that when presented as a distribution alongside the distribution of real samples, these two are indistinguishable by the discriminator. In order to generate different samples with a deterministic generator, the generator receives random numbers as the initial input.

The Math and Technical Details

Typically, for the discriminator we use binary cross entropy loss with label 1 being real and 0 being fake. For the generator, the input is a random vector drawn from a standard normal distribution. Denote the generator by $G_{\phi}(z)$, discriminator by $D_{\theta}(x)$, the distribution of the real samples by $p(x)$ and the input distribution to the generator by $q(z)$. Recall that the binary cross entropy loss with classifier output $y$ and label $\hat{y}$ is

$$L(y, \hat{y}) = -\hat{y} \log y - (1 - \hat{y}) \log (1 - y)$$

For the discriminator, the objective is

$$\min_{\theta} \mathrm{E}_{x \sim p(x)}[L(D_{\theta}(x), 1)] + \mathrm{E}_{z \sim q(z)}[L(D_{\theta}(G_{\phi}(z)), 0)]$$

For the generator, the objective is

$$\max_{\phi} \mathrm{E}_{z \sim q(z)}[L(D_{\theta}(G_{\phi}(z)), 0)]$$

The generator's objective corresponds to maximizing the classification loss of the discriminator on the generated samples. Alternatively, we can minimize the classification loss of the discriminator on the generated samples when labelled as real:

$$\min_{\phi} \mathrm{E}_{z \sim q(z)}[L(D_{\theta}(G_{\phi}(z)), 1)]$$

The strength of the two networks should be balanced, so we train the two networks alternatingly, updating the parameters in both networks once in each interation.

Implementing the GAN

Note: I will be using CIFAR-10 data. You can download it using the following shell script:

wget http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
tar -xzvf cifar-10-python.tar.gz
rm cifar-10-python.tar.gz 

Some helpful functions

In [2]:
def unpickle(file):
    import sys
    if sys.version_info.major == 2:
        import cPickle
        with open(file, 'rb') as fo:
            dict = cPickle.load(fo)
        return dict['data'], dict['labels']
    else:
        import pickle
        with open(file, 'rb') as fo:
            dict = pickle.load(fo, encoding='bytes')
        return dict[b'data'], dict[b'labels']

def load_train_data():
    X = []
    for i in range(5):
        X_, _ = unpickle('data/cifar-10-batches-py/data_batch_%d' % (i + 1))
        X.append(X_)
    X = np.concatenate(X)
    X = X.reshape((X.shape[0], 3, 32, 32)).transpose(0, 2, 3, 1)
    return X

def load_test_data():
    X_, _ = unpickle('data/cifar-10-batches-py/test_batch')
    X = X_.reshape((X_.shape[0], 3, 32, 32)).transpose(0, 2, 3, 1)
    return X

# Load cifar-10 data
train_samples = load_train_data() / 255.0
test_samples = load_test_data() / 255.0
In [3]:
def viz_grid(Xs, padding):
    N, H, W, C = Xs.shape
    grid_size = int(math.ceil(math.sqrt(N)))
    grid_height = H * grid_size + padding * (grid_size + 1)
    grid_width = W * grid_size + padding * (grid_size + 1)
    grid = np.zeros((grid_height, grid_width, C))
    next_idx = 0
    y0, y1 = padding, H + padding
    for y in range(grid_size):
        x0, x1 = padding, W + padding
        for x in range(grid_size):
            if next_idx < N:
                img = Xs[next_idx]
                grid[y0:y1, x0:x1] = img
                next_idx += 1
            x0 += W + padding
            x1 += W + padding
        y0 += H + padding
        y1 += H + padding
    return grid

def set_seed(seed):
    np.random.seed(seed)
    tf.set_random_seed(seed)

def conv2d(input, kernel_size, stride, num_filter, name = 'conv2d'):
    with tf.variable_scope(name):
        stride_shape = [1, stride, stride, 1]
        filter_shape = [kernel_size, kernel_size, input.get_shape()[3], num_filter]

        W = tf.get_variable('w', filter_shape, tf.float32, tf.random_normal_initializer(0.0, 0.02))
        b = tf.get_variable('b', [1, 1, 1, num_filter], initializer = tf.constant_initializer(0.0))
        return tf.nn.conv2d(input, W, stride_shape, padding = 'SAME') + b

def conv2d_transpose(input, kernel_size, stride, num_filter, name = 'conv2d_transpose'):
    with tf.variable_scope(name):
        stride_shape = [1, stride, stride, 1]
        filter_shape = [kernel_size, kernel_size, num_filter, input.get_shape()[3]]
        output_shape = tf.stack([tf.shape(input)[0], tf.shape(input)[1] * 2, tf.shape(input)[2] * 2, num_filter])

        W = tf.get_variable('w', filter_shape, tf.float32, tf.random_normal_initializer(0.0, 0.02))
        b = tf.get_variable('b', [1, 1, 1, num_filter], initializer = tf.constant_initializer(0.0))
        return tf.nn.conv2d_transpose(input, W, output_shape, stride_shape, padding = 'SAME') + b

def fc(input, num_output, name = 'fc'):
    with tf.variable_scope(name):
        num_input = input.get_shape()[1]
        W = tf.get_variable('w', [num_input, num_output], tf.float32, tf.random_normal_initializer(0.0, 0.02))
        b = tf.get_variable('b', [num_output], initializer = tf.constant_initializer(0.0))
        return tf.matmul(input, W) + b

def batch_norm(input, is_training):
    out = tf.contrib.layers.batch_norm(input, decay = 0.99, center = True, scale = True,
                                       is_training = is_training, updates_collections = None)
    return out

def leaky_relu(input, alpha = 0.2):
    return tf.maximum(alpha * input, input)

Implementation Notes

  • _discriminator() and _generator() define the discriminatosr and generator
  • _init_ops() builds the computational graph for the losses
    • fake_samples_op: generate famples from noise
    • dis_loss_op: compute discriminator's loss, with real samples from real_input and fake samples generated by the generator
    • gen_loss_op: compute generator's loss
  • dis_train_op and gen_train_op define the optimization functions:

    • Uses RMSprop for training (Adam performs poorly with an unstable objective as is the case in GANs)
    • Rather than updating all the parameters all the time for both the generator and discriminator simultaneously, when training one network, the other one kept fixed
  • The batch normalization layers should operate in training mode. As per How to Train a GAN? Tips and tricks to make GANs work, real samples and fake samples are put in different batches when training the discriminator.

In [24]:
class DCGAN(object):

    def __init__(self):
        self.num_epoch = 25
        self.batch_size = 32
        self.log_step = 50
        self.visualize_step = 200
        self.code_size = 64
        self.learning_rate = 1e-4
        self.vis_learning_rate = 1e-2
        self.recon_steps = 100
        self.actmax_steps = 100
        
        self._dis_called = False
        self._gen_called = False

        self.tracked_noise = np.random.normal(0, 1, [64, self.code_size])

        self.real_input = tf.placeholder(tf.float32, [None, 32, 32, 3])
        self.real_label = tf.placeholder(tf.float32, [None, 1])
        self.fake_label = tf.placeholder(tf.float32, [None, 1])
        self.noise = tf.placeholder(tf.float32, [None, self.code_size])
        
        self.is_train = tf.placeholder(tf.bool)
        
        self.recon_sample = tf.placeholder(tf.float32, [1, 32, 32, 3])
        self.actmax_label = tf.placeholder(tf.float32, [1, 1])
        
        with tf.variable_scope('actmax'):
            self.actmax_code = tf.get_variable('actmax_code', [1, self.code_size],
                                               initializer = tf.constant_initializer(0.0))
        
        self._init_ops()

    def _discriminator(self, input):
        # We have multiple instances of the discriminator in the same computation graph,
        # so set variable sharing if this is not the first invocation of this function.
        with tf.variable_scope('dis', reuse = self._dis_called):
            self._dis_called = True
            dis_conv1 = conv2d(input, 4, 2, 32, 'conv1')
            dis_lrelu1 = leaky_relu(dis_conv1)
            dis_conv2 = conv2d(dis_lrelu1, 4, 2, 64, 'conv2')
            dis_batchnorm2 = batch_norm(dis_conv2, self.is_train)
            dis_lrelu2 = leaky_relu(dis_batchnorm2)
            dis_conv3 = conv2d(dis_lrelu2, 4, 2, 128, 'conv3')
            dis_batchnorm3 = batch_norm(dis_conv3, self.is_train)
            dis_lrelu3 = leaky_relu(dis_batchnorm3)
            dis_reshape3 = tf.reshape(dis_lrelu3, [-1, 4 * 4 * 128])
            dis_fc4 = fc(dis_reshape3, 1, 'fc4')
            return dis_fc4

    def _generator(self, input):
        with tf.variable_scope('gen', reuse = self._gen_called):
            self._gen_called = True
            gen_fc1 = fc(input, 4 * 4 * 128, 'fc1')
            gen_reshape1 = tf.reshape(gen_fc1, [-1, 4, 4, 128])
            gen_batchnorm1 = batch_norm(gen_reshape1, self.is_train)
            gen_lrelu1 = leaky_relu(gen_batchnorm1)
            gen_conv2 = conv2d_transpose(gen_lrelu1, 4, 2, 64, 'conv2')
            gen_batchnorm2 = batch_norm(gen_conv2, self.is_train)
            gen_lrelu2 = leaky_relu(gen_batchnorm2)
            gen_conv3 = conv2d_transpose(gen_lrelu2, 4, 2, 32, 'conv3')
            gen_batchnorm3 = batch_norm(gen_conv3, self.is_train)
            gen_lrelu3 = leaky_relu(gen_batchnorm3)
            gen_conv4 = conv2d_transpose(gen_lrelu3, 4, 2, 3, 'conv4')
            gen_sigmoid4 = tf.sigmoid(gen_conv4)
            return gen_sigmoid4

    def _loss(self, labels, logits):
        loss = tf.nn.sigmoid_cross_entropy_with_logits(labels = labels, logits = logits)
        return tf.reduce_mean(loss)

    def _reconstruction_loss(self, generated, target):
        loss = tf.nn.l2_loss(generated - target)
        return tf.reduce_mean(loss)
    
    # Define operations
    def _init_ops(self):
        
        # generate famples from noise
        self.fake_samples_op = self._generator(self.noise)
        
        # compute discriminator's loss, with real samples from real_input and fake samples generated by the generator
        self.dis_loss_op=self._loss(self.real_label, self._discriminator(self.real_input)) + self._loss(self.fake_label,self._discriminator(self.fake_samples_op))

        #compute generator's loss
        self.gen_loss_op= self._loss(self.real_label,self._discriminator(self.fake_samples_op))

        dis_optimizer = tf.train.RMSPropOptimizer(self.learning_rate)
        dis_train_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES,
                                             "dis")
        self.dis_train_op = dis_optimizer.minimize(self.dis_loss_op, var_list=dis_train_vars)

        gen_optimizer = tf.train.RMSPropOptimizer(self.learning_rate)
        gen_train_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES,
                                              "gen")                     
        self.gen_train_op = gen_optimizer.minimize(self.gen_loss_op, var_list=gen_train_vars)
    
        
        self.actmax_sample_op = self._generator(self.actmax_code)
        actmax_dis = self._discriminator(self.actmax_sample_op)
        self.actmax_loss_op = self._loss(self.actmax_label, actmax_dis)

        actmax_optimizer = tf.train.AdamOptimizer(self.vis_learning_rate)
        self.actmax_op = actmax_optimizer.minimize(self.actmax_loss_op, var_list = [self.actmax_code])
        
        self.recon_loss_op = self._reconstruction_loss(self.actmax_sample_op, self.recon_sample) #generated, target

        recon_train_vars = tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES,"actmax")     
        
        recon_optimizer = tf.train.AdamOptimizer(self.vis_learning_rate)

        self.reconstruct_op = recon_optimizer.minimize(self.recon_loss_op,var_list = [self.actmax_code])

    # Training function
    def train(self, sess, train_samples):
        sess.run(tf.global_variables_initializer())

        num_train = train_samples.shape[0]
        step = 0
        
        # smooth the loss curve so that it does not fluctuate too much
        smooth_factor = 0.95
        plot_dis_s = 0
        plot_gen_s = 0
        plot_ws = 0
        
        dis_losses = []
        gen_losses = []
        max_steps = int(self.num_epoch * (num_train // self.batch_size))
        print('Start training ...')
        for epoch in range(self.num_epoch):
            for i in range(num_train // self.batch_size):
                step += 1

                batch_samples = train_samples[i * self.batch_size : (i + 1) * self.batch_size]
                noise = np.random.normal(0, 1, [self.batch_size, self.code_size])
                zeros = np.zeros([self.batch_size, 1])
                ones = np.ones([self.batch_size, 1])
        
                # feed dictionary 
                dis_feed_dict = {self.is_train: self._dis_called, self.real_input:batch_samples, self.noise:noise, self.real_label: ones, self.fake_label:zeros}
                _, dis_loss = sess.run([self.dis_train_op, self.dis_loss_op], feed_dict = dis_feed_dict)

                gen_feed_dict = {self.is_train: self._gen_called, self.real_input:batch_samples, self.noise:noise, self.real_label: ones, self.fake_label:zeros}
                _, gen_loss = sess.run([self.gen_train_op, self.gen_loss_op], feed_dict = gen_feed_dict)

                plot_dis_s = plot_dis_s * smooth_factor + dis_loss * (1 - smooth_factor)
                plot_gen_s = plot_gen_s * smooth_factor + gen_loss * (1 - smooth_factor)
                plot_ws = plot_ws * smooth_factor + (1 - smooth_factor)
                dis_losses.append(plot_dis_s / plot_ws)
                gen_losses.append(plot_gen_s / plot_ws)

                if step % self.log_step == 0:
                    print('Iteration {0}/{1}: dis loss = {2:.4f}, gen loss = {3:.4f}'.format(step, max_steps, dis_loss, gen_loss))

            fig = plt.figure(figsize = (8, 8))   
            ax1 = plt.subplot(111)
            ax1.imshow(viz_grid(self.generate(self.tracked_noise), 1))
            plt.show()

            plt.plot(dis_losses)
            plt.title('discriminator loss')
            plt.xlabel('iterations')
            plt.ylabel('loss')
            plt.show()

            plt.plot(gen_losses)
            plt.title('generator loss')
            plt.xlabel('iterations')
            plt.ylabel('loss')
            plt.show()
        print('... Done!')

    # Find the reconstruction of one input sample
    def reconstruct_one_sample(self, sample):
        
        # Initialize 
        actmax_init_val = np.zeros([1,self.code_size])
        
        sess.run(self.actmax_code.assign(actmax_init_val))
        last_reconstruction = None
        last_loss = None
        for i in range(self.recon_steps):
        
            # Feed dict
            recon_feed_dict = { self.recon_sample: sample, self.is_train: False}
            
            run_ops = [self.recon_loss_op, self.reconstruct_op, self.actmax_sample_op]
            last_loss, _, last_reconstruction = sess.run(run_ops, feed_dict = recon_feed_dict)
        return last_loss, last_reconstruction

    # Find the reconstruction of a batch of samples
    def reconstruct(self, samples):
        reconstructions = np.zeros(samples.shape)
        total_loss = 0
        for i in range(samples.shape[0]):
            loss, reconstructions[i:i+1] = self.reconstruct_one_sample(samples[i:i+1])
            print(loss)
            total_loss += loss
        return total_loss / samples.shape[0], reconstructions

    # Generates a single sample from input code
    def generate_one_sample(self, code):
        
        # Feed dict
        gen_vis_feed_dict = {self.noise:code, self.is_train: self._gen_called}

        generated = sess.run(self.fake_samples_op, feed_dict = gen_vis_feed_dict)
        return generated

    # Generates samples from input batch of codes
    def generate(self, codes):
        generated = np.zeros((codes.shape[0], 32, 32, 3))
        for i in range(codes.shape[0]):
            generated[i:i+1] = self.generate_one_sample(codes[i:i+1])
        return generated

    # Perform activation maximization on one initial code
    def actmax_one_sample(self, initial_code):
        
        actmax_init_val = tf.convert_to_tensor(initial_code, dtype = tf.float32)
        sess.run(self.actmax_code.assign(actmax_init_val))
        for i in range(self.actmax_steps):
            actmax_feed_dict = {self.actmax_label: np.ones([1, 1]),self.is_train: False}
            _, last_actmax = sess.run([self.actmax_op, self.actmax_sample_op], feed_dict = actmax_feed_dict)
        return last_actmax

    # Perform activation maximization on a batch of different initial codes
    def actmax(self, initial_codes):
        actmax_results = np.zeros((initial_codes.shape[0], 32, 32, 3))
        for i in range(initial_codes.shape[0]):
            actmax_results[i:i+1] = self.actmax_one_sample(initial_codes[i:i+1])
        return actmax_results.clip(0, 1)

Now let's do the training!

In [5]:
tf.reset_default_graph()
set_seed(21)

with tf.Session() as sess:
    with tf.device('/cpu:0'):
        dcgan = DCGAN()
        sess.run(tf.global_variables_initializer())
        dcgan.train(sess, train_samples)
        dis_var_list = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, 'dis')
        gen_var_list = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, 'gen')
        saver = tf.train.Saver(dis_var_list + gen_var_list)
        saver.save(sess, 'model/dcgan')
  • Start training ...
    Start training ...
    Iteration 50/39050: dis loss = 0.4689, gen loss = 2.1449
    Iteration 100/39050: dis loss = 0.0753, gen loss = 4.0175
    Iteration 150/39050: dis loss = 0.1867, gen loss = 5.2755
    Iteration 200/39050: dis loss = 0.1900, gen loss = 2.8303
    Iteration 250/39050: dis loss = 0.0443, gen loss = 4.2992
    Iteration 300/39050: dis loss = 0.3052, gen loss = 1.2024
    Iteration 350/39050: dis loss = 0.6522, gen loss = 1.3441
    Iteration 400/39050: dis loss = 0.5761, gen loss = 1.7183
    Iteration 450/39050: dis loss = 0.6572, gen loss = 1.5509
    Iteration 500/39050: dis loss = 0.5829, gen loss = 3.0608
    Iteration 550/39050: dis loss = 0.8584, gen loss = 1.6992
    Iteration 600/39050: dis loss = 0.7362, gen loss = 1.9687
    Iteration 650/39050: dis loss = 1.0542, gen loss = 2.2370
    Iteration 700/39050: dis loss = 0.7710, gen loss = 1.0471
    Iteration 750/39050: dis loss = 0.7598, gen loss = 2.0690
    Iteration 800/39050: dis loss = 0.5842, gen loss = 2.4567
    Iteration 850/39050: dis loss = 0.7619, gen loss = 1.4316
    Iteration 900/39050: dis loss = 0.7883, gen loss = 0.7142
    Iteration 950/39050: dis loss = 0.7357, gen loss = 0.8942
    Iteration 1000/39050: dis loss = 1.0445, gen loss = 1.6115
    Iteration 1050/39050: dis loss = 1.0869, gen loss = 0.9454
    Iteration 1100/39050: dis loss = 1.3355, gen loss = 0.5997
    Iteration 1150/39050: dis loss = 1.8223, gen loss = 0.3788
    Iteration 1200/39050: dis loss = 1.5696, gen loss = 0.9683
    Iteration 1250/39050: dis loss = 1.2392, gen loss = 1.2401
    Iteration 1300/39050: dis loss = 1.3231, gen loss = 0.8438
    Iteration 1350/39050: dis loss = 1.2808, gen loss = 0.7731
    Iteration 1400/39050: dis loss = 1.2577, gen loss = 0.7928
    Iteration 1450/39050: dis loss = 1.2482, gen loss = 1.1351
    Iteration 1500/39050: dis loss = 1.4111, gen loss = 0.7686
    Iteration 1550/39050: dis loss = 1.2577, gen loss = 1.3398
    
Iteration 1600/39050: dis loss = 1.2031, gen loss = 1.1511
Iteration 1650/39050: dis loss = 1.2184, gen loss = 1.0842
Iteration 1700/39050: dis loss = 1.1110, gen loss = 1.2085
Iteration 1750/39050: dis loss = 1.0987, gen loss = 0.8692
Iteration 1800/39050: dis loss = 1.2760, gen loss = 0.9083
Iteration 1850/39050: dis loss = 1.4648, gen loss = 0.5743
Iteration 1900/39050: dis loss = 1.1157, gen loss = 1.2322
Iteration 1950/39050: dis loss = 1.1252, gen loss = 1.2072
Iteration 2000/39050: dis loss = 1.2159, gen loss = 1.3421
Iteration 2050/39050: dis loss = 1.1546, gen loss = 1.1153
Iteration 2100/39050: dis loss = 1.3095, gen loss = 0.6509
Iteration 2150/39050: dis loss = 1.2638, gen loss = 0.7121
Iteration 2200/39050: dis loss = 1.1688, gen loss = 1.0684
Iteration 2250/39050: dis loss = 1.3496, gen loss = 0.6672
Iteration 2300/39050: dis loss = 1.4108, gen loss = 0.5497
Iteration 2350/39050: dis loss = 1.1287, gen loss = 0.8740
Iteration 2400/39050: dis loss = 1.2108, gen loss = 0.7311
Iteration 2450/39050: dis loss = 1.2838, gen loss = 0.8645
Iteration 2500/39050: dis loss = 1.2155, gen loss = 0.7506
Iteration 2550/39050: dis loss = 1.3598, gen loss = 0.6920
Iteration 2600/39050: dis loss = 1.4037, gen loss = 1.7740
Iteration 2650/39050: dis loss = 1.6226, gen loss = 0.3765
Iteration 2700/39050: dis loss = 1.0695, gen loss = 0.7675
Iteration 2750/39050: dis loss = 0.9744, gen loss = 1.1849
Iteration 2800/39050: dis loss = 1.1708, gen loss = 1.1818
Iteration 2850/39050: dis loss = 1.1385, gen loss = 0.5871
Iteration 2900/39050: dis loss = 1.0116, gen loss = 1.4025
Iteration 2950/39050: dis loss = 1.0804, gen loss = 0.8710
Iteration 3000/39050: dis loss = 1.1865, gen loss = 0.5384
Iteration 3050/39050: dis loss = 0.8199, gen loss = 1.7396
Iteration 3100/39050: dis loss = 0.8251, gen loss = 1.5662
Iteration 3150/39050: dis loss = 1.0027, gen loss = 1.0165
Iteration 3200/39050: dis loss = 1.0980, gen loss = 1.5926
Iteration 3250/39050: dis loss = 0.7990, gen loss = 1.3859
Iteration 3300/39050: dis loss = 0.6716, gen loss = 1.2904
Iteration 3350/39050: dis loss = 0.9689, gen loss = 0.8724
Iteration 3400/39050: dis loss = 0.9866, gen loss = 1.5589
Iteration 3450/39050: dis loss = 0.8250, gen loss = 1.1016
Iteration 3500/39050: dis loss = 0.8421, gen loss = 1.6547
Iteration 3550/39050: dis loss = 0.8940, gen loss = 1.6957
Iteration 3600/39050: dis loss = 0.9957, gen loss = 0.6294
Iteration 3650/39050: dis loss = 0.8493, gen loss = 1.5579
Iteration 3700/39050: dis loss = 0.9191, gen loss = 1.5726
Iteration 3750/39050: dis loss = 1.1288, gen loss = 1.8221
Iteration 3800/39050: dis loss = 0.8957, gen loss = 1.4166
Iteration 3850/39050: dis loss = 0.9192, gen loss = 1.5794
Iteration 3900/39050: dis loss = 0.9474, gen loss = 0.8425
Iteration 3950/39050: dis loss = 0.8337, gen loss = 1.5037
Iteration 4000/39050: dis loss = 0.9289, gen loss = 0.9223
Iteration 4050/39050: dis loss = 1.1132, gen loss = 0.8068
Iteration 4100/39050: dis loss = 0.8830, gen loss = 1.8514
Iteration 4150/39050: dis loss = 1.0255, gen loss = 0.8617
Iteration 4200/39050: dis loss = 1.1613, gen loss = 0.5536
Iteration 4250/39050: dis loss = 0.7410, gen loss = 1.2701
Iteration 4300/39050: dis loss = 0.7837, gen loss = 1.1168
Iteration 4350/39050: dis loss = 1.3113, gen loss = 1.7652
Iteration 4400/39050: dis loss = 1.0218, gen loss = 1.3508
Iteration 4450/39050: dis loss = 0.8823, gen loss = 1.3786
Iteration 4500/39050: dis loss = 1.6240, gen loss = 2.4969
Iteration 4550/39050: dis loss = 1.0021, gen loss = 1.4473
Iteration 4600/39050: dis loss = 0.7741, gen loss = 1.7860
Iteration 4650/39050: dis loss = 1.1113, gen loss = 0.7978
  • Iteration 4700/39050: dis loss = 0.9156, gen loss = 0.9124
    Iteration 4750/39050: dis loss = 1.1209, gen loss = 1.1502
    Iteration 4750/39050: dis loss = 1.1209, gen loss = 1.1502
    Iteration 4800/39050: dis loss = 1.0158, gen loss = 0.8734
    Iteration 4850/39050: dis loss = 0.6890, gen loss = 1.2818
    Iteration 4900/39050: dis loss = 0.7881, gen loss = 1.0831
    Iteration 4950/39050: dis loss = 0.7392, gen loss = 1.3236
    Iteration 5000/39050: dis loss = 0.7877, gen loss = 1.0785
    Iteration 5050/39050: dis loss = 0.9259, gen loss = 0.9991
    Iteration 5100/39050: dis loss = 0.7126, gen loss = 1.1967
    Iteration 5150/39050: dis loss = 1.1787, gen loss = 0.6404
    Iteration 5200/39050: dis loss = 0.9089, gen loss = 1.5375
    Iteration 5250/39050: dis loss = 0.9207, gen loss = 1.9856
    Iteration 5300/39050: dis loss = 0.7816, gen loss = 2.0516
    Iteration 5350/39050: dis loss = 0.8388, gen loss = 0.9700
    Iteration 5400/39050: dis loss = 0.6002, gen loss = 1.3514
    Iteration 5450/39050: dis loss = 0.8374, gen loss = 1.2595
    Iteration 5500/39050: dis loss = 0.9832, gen loss = 1.8020
    Iteration 5550/39050: dis loss = 0.9698, gen loss = 0.8401
    Iteration 5600/39050: dis loss = 0.7800, gen loss = 1.8720
    Iteration 5650/39050: dis loss = 1.1439, gen loss = 2.3567
    Iteration 5700/39050: dis loss = 0.9037, gen loss = 1.2299
    Iteration 5750/39050: dis loss = 0.7217, gen loss = 1.2329
    Iteration 5800/39050: dis loss = 0.8840, gen loss = 1.3790
    Iteration 5850/39050: dis loss = 0.8157, gen loss = 1.0814
    Iteration 5900/39050: dis loss = 0.8721, gen loss = 0.9906
    Iteration 5950/39050: dis loss = 0.9045, gen loss = 2.2106
    Iteration 6000/39050: dis loss = 0.8494, gen loss = 0.9732
    Iteration 6050/39050: dis loss = 0.9089, gen loss = 1.1907
    Iteration 6100/39050: dis loss = 1.0058, gen loss = 1.7990
    Iteration 6150/39050: dis loss = 0.9023, gen loss = 0.9796
    Iteration 6200/39050: dis loss = 0.8674, gen loss = 1.7555
    
  • Iteration 6250/39050: dis loss = 0.8392, gen loss = 1.7999
    Iteration 6300/39050: dis loss = 1.0095, gen loss = 1.5345
    Iteration 6300/39050: dis loss = 1.0095, gen loss = 1.5345
    Iteration 6350/39050: dis loss = 1.1529, gen loss = 0.5430
    Iteration 6400/39050: dis loss = 0.8658, gen loss = 1.6714
    Iteration 6450/39050: dis loss = 0.9974, gen loss = 0.8458
    Iteration 6500/39050: dis loss = 0.9295, gen loss = 1.6889
    Iteration 6550/39050: dis loss = 1.0774, gen loss = 0.5026
    Iteration 6600/39050: dis loss = 0.7836, gen loss = 1.0435
    Iteration 6650/39050: dis loss = 1.1484, gen loss = 2.2252
    Iteration 6700/39050: dis loss = 0.8224, gen loss = 1.9947
    Iteration 6750/39050: dis loss = 1.0800, gen loss = 0.6593
    Iteration 6800/39050: dis loss = 0.8839, gen loss = 1.5823
    Iteration 6850/39050: dis loss = 0.9833, gen loss = 2.0279
    Iteration 6900/39050: dis loss = 1.2375, gen loss = 0.5519
    Iteration 6950/39050: dis loss = 0.6858, gen loss = 1.6787
    Iteration 7000/39050: dis loss = 0.7746, gen loss = 1.4491
    Iteration 7050/39050: dis loss = 0.9896, gen loss = 1.0547
    Iteration 7100/39050: dis loss = 0.9416, gen loss = 1.1636
    Iteration 7150/39050: dis loss = 0.8479, gen loss = 1.9848
    Iteration 7200/39050: dis loss = 1.3160, gen loss = 1.5777
    Iteration 7250/39050: dis loss = 1.3823, gen loss = 2.4394
    Iteration 7300/39050: dis loss = 1.0277, gen loss = 1.5580
    Iteration 7350/39050: dis loss = 0.7094, gen loss = 1.8452
    Iteration 7400/39050: dis loss = 0.8971, gen loss = 1.9397
    Iteration 7450/39050: dis loss = 0.8585, gen loss = 1.3047
    Iteration 7500/39050: dis loss = 1.1674, gen loss = 1.7106
    Iteration 7550/39050: dis loss = 1.0286, gen loss = 1.8812
    Iteration 7600/39050: dis loss = 0.9319, gen loss = 2.5818
    Iteration 7650/39050: dis loss = 1.1555, gen loss = 1.6398
    Iteration 7700/39050: dis loss = 0.9222, gen loss = 1.4755
    Iteration 7750/39050: dis loss = 0.8637, gen loss = 1.9028
    Iteration 7800/39050: dis loss = 0.9316, gen loss = 1.7964
    
  • Iteration 7850/39050: dis loss = 1.0384, gen loss = 0.7977
    Iteration 7900/39050: dis loss = 0.7976, gen loss = 2.3147
    Iteration 7900/39050: dis loss = 0.7976, gen loss = 2.3147
    Iteration 7950/39050: dis loss = 0.7788, gen loss = 1.7705
    Iteration 8000/39050: dis loss = 0.8205, gen loss = 2.0652
    Iteration 8050/39050: dis loss = 0.8822, gen loss = 2.0503
    Iteration 8100/39050: dis loss = 1.0025, gen loss = 1.6417
    Iteration 8150/39050: dis loss = 1.0212, gen loss = 1.7440
    Iteration 8200/39050: dis loss = 0.9420, gen loss = 1.7864
    Iteration 8250/39050: dis loss = 0.9454, gen loss = 0.5508
    Iteration 8300/39050: dis loss = 1.7398, gen loss = 2.2462
    Iteration 8350/39050: dis loss = 0.9475, gen loss = 0.7772
    Iteration 8400/39050: dis loss = 0.8759, gen loss = 1.1854
    Iteration 8450/39050: dis loss = 0.8385, gen loss = 1.7827
    Iteration 8500/39050: dis loss = 0.9271, gen loss = 0.9242
    Iteration 8550/39050: dis loss = 0.8655, gen loss = 1.1149
    Iteration 8600/39050: dis loss = 1.0252, gen loss = 0.7329
    Iteration 8650/39050: dis loss = 1.1508, gen loss = 0.6422
    Iteration 8700/39050: dis loss = 0.9238, gen loss = 0.9891
    Iteration 8750/39050: dis loss = 0.6923, gen loss = 1.4707
    Iteration 8800/39050: dis loss = 0.6468, gen loss = 1.4686
    Iteration 8850/39050: dis loss = 0.8811, gen loss = 0.6909
    Iteration 8900/39050: dis loss = 0.7941, gen loss = 1.1111
    Iteration 8950/39050: dis loss = 0.9615, gen loss = 0.6183
    Iteration 9000/39050: dis loss = 0.8241, gen loss = 1.3892
    Iteration 9050/39050: dis loss = 0.9315, gen loss = 2.2555
    Iteration 9100/39050: dis loss = 0.8817, gen loss = 1.6336
    Iteration 9150/39050: dis loss = 1.3844, gen loss = 2.2838
    Iteration 9200/39050: dis loss = 0.8150, gen loss = 1.5051
    Iteration 9250/39050: dis loss = 1.7697, gen loss = 0.3776
    Iteration 9300/39050: dis loss = 0.8499, gen loss = 0.8646
    Iteration 9350/39050: dis loss = 0.6899, gen loss = 1.7437
    
  • Iteration 9400/39050: dis loss = 0.9667, gen loss = 0.9384
    Iteration 9450/39050: dis loss = 0.9321, gen loss = 1.3871
    Iteration 9450/39050: dis loss = 0.9321, gen loss = 1.3871
    Iteration 9500/39050: dis loss = 0.8464, gen loss = 1.4878
    Iteration 9550/39050: dis loss = 1.1002, gen loss = 2.1031
    Iteration 9600/39050: dis loss = 0.7242, gen loss = 1.0136
    Iteration 9650/39050: dis loss = 0.9480, gen loss = 0.8542
    Iteration 9700/39050: dis loss = 0.8337, gen loss = 0.7955
    Iteration 9750/39050: dis loss = 0.6858, gen loss = 1.2641
    Iteration 9800/39050: dis loss = 0.9860, gen loss = 2.4572
    Iteration 9850/39050: dis loss = 0.8347, gen loss = 1.1236
    Iteration 9900/39050: dis loss = 1.0620, gen loss = 2.2756
    Iteration 9950/39050: dis loss = 1.1436, gen loss = 0.7265
    Iteration 10000/39050: dis loss = 0.9586, gen loss = 0.9559
    Iteration 10050/39050: dis loss = 1.1765, gen loss = 0.6047
    Iteration 10100/39050: dis loss = 0.9908, gen loss = 2.0065
    Iteration 10150/39050: dis loss = 0.8704, gen loss = 1.7737
    Iteration 10200/39050: dis loss = 1.1506, gen loss = 2.4130
    Iteration 10250/39050: dis loss = 0.7920, gen loss = 2.0156
    Iteration 10300/39050: dis loss = 0.9897, gen loss = 0.9061
    Iteration 10350/39050: dis loss = 0.9876, gen loss = 1.7795
    Iteration 10400/39050: dis loss = 1.2502, gen loss = 2.4404
    Iteration 10450/39050: dis loss = 0.9508, gen loss = 0.8594
    Iteration 10500/39050: dis loss = 0.9435, gen loss = 1.0807
    Iteration 10550/39050: dis loss = 0.8695, gen loss = 0.5708
    Iteration 10600/39050: dis loss = 0.5017, gen loss = 1.8943
    Iteration 10650/39050: dis loss = 1.0830, gen loss = 1.8701
    Iteration 10700/39050: dis loss = 0.6583, gen loss = 1.7120
    Iteration 10750/39050: dis loss = 0.8471, gen loss = 2.0408
    Iteration 10800/39050: dis loss = 0.8889, gen loss = 1.6001
    Iteration 10850/39050: dis loss = 0.8381, gen loss = 1.8784
    Iteration 10900/39050: dis loss = 0.7850, gen loss = 1.1475
    
  • Iteration 10950/39050: dis loss = 0.8323, gen loss = 0.7931
    Iteration 11000/39050: dis loss = 0.9270, gen loss = 0.7871
    Iteration 11000/39050: dis loss = 0.9270, gen loss = 0.7871
    Iteration 11050/39050: dis loss = 1.0136, gen loss = 0.7502
    Iteration 11100/39050: dis loss = 0.7993, gen loss = 1.6817
    Iteration 11150/39050: dis loss = 1.0331, gen loss = 0.8764
    Iteration 11200/39050: dis loss = 0.8709, gen loss = 1.0618
    Iteration 11250/39050: dis loss = 0.9730, gen loss = 0.8838
    Iteration 11300/39050: dis loss = 1.0351, gen loss = 0.9564
    Iteration 11350/39050: dis loss = 0.7642, gen loss = 2.0513
    Iteration 11400/39050: dis loss = 0.8556, gen loss = 1.3378
    Iteration 11450/39050: dis loss = 1.8850, gen loss = 0.6310
    Iteration 11500/39050: dis loss = 1.3707, gen loss = 2.5737
    Iteration 11550/39050: dis loss = 1.1818, gen loss = 1.9684
    Iteration 11600/39050: dis loss = 0.9653, gen loss = 2.2239
    Iteration 11650/39050: dis loss = 0.9708, gen loss = 0.7931
    Iteration 11700/39050: dis loss = 1.0132, gen loss = 0.5650
    Iteration 11750/39050: dis loss = 0.6983, gen loss = 1.6254
    Iteration 11800/39050: dis loss = 0.9924, gen loss = 2.0519
    Iteration 11850/39050: dis loss = 0.7976, gen loss = 1.7033
    Iteration 11900/39050: dis loss = 0.8316, gen loss = 1.4316
    Iteration 11950/39050: dis loss = 0.8457, gen loss = 2.0149
    Iteration 12000/39050: dis loss = 0.8171, gen loss = 2.1146
    Iteration 12050/39050: dis loss = 0.9120, gen loss = 1.2544
    Iteration 12100/39050: dis loss = 0.9331, gen loss = 2.2833
    Iteration 12150/39050: dis loss = 0.7650, gen loss = 1.8031
    Iteration 12200/39050: dis loss = 0.8572, gen loss = 0.9771
    Iteration 12250/39050: dis loss = 0.6618, gen loss = 1.4070
    Iteration 12300/39050: dis loss = 0.7956, gen loss = 1.1819
    Iteration 12350/39050: dis loss = 0.9445, gen loss = 1.0443
    Iteration 12400/39050: dis loss = 0.8622, gen loss = 1.3810
    Iteration 12450/39050: dis loss = 1.0979, gen loss = 1.9867
    
  • Iteration 12500/39050: dis loss = 0.8043, gen loss = 2.3140
    Iteration 12550/39050: dis loss = 0.9080, gen loss = 0.8824
    Iteration 12550/39050: dis loss = 0.9080, gen loss = 0.8824
    Iteration 12600/39050: dis loss = 0.8341, gen loss = 1.0834
    Iteration 12650/39050: dis loss = 1.2138, gen loss = 2.2596
    Iteration 12700/39050: dis loss = 1.1021, gen loss = 2.0770
    Iteration 12750/39050: dis loss = 0.7685, gen loss = 1.1576
    Iteration 12800/39050: dis loss = 1.1617, gen loss = 2.0896
    Iteration 12850/39050: dis loss = 0.7655, gen loss = 1.8873
    Iteration 12900/39050: dis loss = 0.7501, gen loss = 2.2012
    Iteration 12950/39050: dis loss = 0.7378, gen loss = 1.4543
    Iteration 13000/39050: dis loss = 0.9605, gen loss = 0.9031
    Iteration 13050/39050: dis loss = 1.1241, gen loss = 2.2983
    Iteration 13100/39050: dis loss = 0.7354, gen loss = 1.8881
    Iteration 13150/39050: dis loss = 1.0097, gen loss = 1.0462
    Iteration 13200/39050: dis loss = 0.7250, gen loss = 1.4289
    Iteration 13250/39050: dis loss = 0.9557, gen loss = 1.5815
    Iteration 13300/39050: dis loss = 1.2612, gen loss = 0.7615
    Iteration 13350/39050: dis loss = 1.1179, gen loss = 0.7884
    Iteration 13400/39050: dis loss = 0.9846, gen loss = 1.0815
    Iteration 13450/39050: dis loss = 0.8344, gen loss = 1.2651
    Iteration 13500/39050: dis loss = 1.1393, gen loss = 1.5004
    Iteration 13550/39050: dis loss = 0.9484, gen loss = 1.5769
    Iteration 13600/39050: dis loss = 1.0491, gen loss = 1.9598
    Iteration 13650/39050: dis loss = 0.9571, gen loss = 1.8140
    Iteration 13700/39050: dis loss = 1.1591, gen loss = 1.5366
    Iteration 13750/39050: dis loss = 1.1430, gen loss = 1.2375
    Iteration 13800/39050: dis loss = 0.8564, gen loss = 1.1143
    Iteration 13850/39050: dis loss = 0.6445, gen loss = 1.9608
    Iteration 13900/39050: dis loss = 0.9660, gen loss = 1.1238
    Iteration 13950/39050: dis loss = 1.0542, gen loss = 0.8031
    Iteration 14000/39050: dis loss = 0.7560, gen loss = 1.6010
    Iteration 14050/39050: dis loss = 1.1872, gen loss = 1.9454
    
  • Iteration 14100/39050: dis loss = 0.9174, gen loss = 1.2988
    Iteration 14150/39050: dis loss = 0.9222, gen loss = 1.2593
    Iteration 14150/39050: dis loss = 0.9222, gen loss = 1.2593
    Iteration 14200/39050: dis loss = 0.9226, gen loss = 1.5954
    Iteration 14250/39050: dis loss = 1.0339, gen loss = 1.9590
    Iteration 14300/39050: dis loss = 0.9182, gen loss = 1.1577
    Iteration 14350/39050: dis loss = 0.7512, gen loss = 0.9127
    Iteration 14400/39050: dis loss = 0.9355, gen loss = 0.9812
    Iteration 14450/39050: dis loss = 1.2172, gen loss = 1.4339
    Iteration 14500/39050: dis loss = 1.8363, gen loss = 2.2567
    Iteration 14550/39050: dis loss = 0.9986, gen loss = 1.4912
    Iteration 14600/39050: dis loss = 1.1753, gen loss = 0.6898
    Iteration 14650/39050: dis loss = 0.9669, gen loss = 0.6557
    Iteration 14700/39050: dis loss = 1.1165, gen loss = 0.8989
    Iteration 14750/39050: dis loss = 1.3666, gen loss = 0.6498
    Iteration 14800/39050: dis loss = 1.0359, gen loss = 1.0561
    Iteration 14850/39050: dis loss = 0.6814, gen loss = 1.5458
    Iteration 14900/39050: dis loss = 1.4619, gen loss = 0.6002
    Iteration 14950/39050: dis loss = 0.9960, gen loss = 1.8946
    Iteration 15000/39050: dis loss = 0.8243, gen loss = 1.6158
    Iteration 15050/39050: dis loss = 0.8757, gen loss = 2.0366
    Iteration 15100/39050: dis loss = 1.2802, gen loss = 0.6871
    Iteration 15150/39050: dis loss = 0.8842, gen loss = 1.0018
    Iteration 15200/39050: dis loss = 1.0811, gen loss = 1.3669
    Iteration 15250/39050: dis loss = 0.9203, gen loss = 1.9054
    Iteration 15300/39050: dis loss = 0.8338, gen loss = 1.4129
    Iteration 15350/39050: dis loss = 1.0934, gen loss = 0.7985
    Iteration 15400/39050: dis loss = 1.1074, gen loss = 1.5960
    Iteration 15450/39050: dis loss = 1.2418, gen loss = 0.7396
    Iteration 15500/39050: dis loss = 0.9533, gen loss = 1.1088
    Iteration 15550/39050: dis loss = 0.9078, gen loss = 1.4285
    Iteration 15600/39050: dis loss = 0.9713, gen loss = 2.2780
    
  • Iteration 15650/39050: dis loss = 1.0733, gen loss = 0.9157
    Iteration 15700/39050: dis loss = 0.6468, gen loss = 1.6225
    Iteration 15700/39050: dis loss = 0.6468, gen loss = 1.6225
    Iteration 15750/39050: dis loss = 0.7302, gen loss = 1.1504
    Iteration 15800/39050: dis loss = 1.1546, gen loss = 0.8431
    Iteration 15850/39050: dis loss = 0.9669, gen loss = 1.0811
    Iteration 15900/39050: dis loss = 1.0263, gen loss = 2.5202
    Iteration 15950/39050: dis loss = 0.8079, gen loss = 2.0426
    Iteration 16000/39050: dis loss = 1.0454, gen loss = 1.9760
    Iteration 16050/39050: dis loss = 0.9248, gen loss = 1.6849
    Iteration 16100/39050: dis loss = 0.7886, gen loss = 2.3837
    Iteration 16150/39050: dis loss = 1.5321, gen loss = 2.2804
    Iteration 16200/39050: dis loss = 1.2953, gen loss = 0.6558
    Iteration 16250/39050: dis loss = 1.0072, gen loss = 1.0144
    Iteration 16300/39050: dis loss = 1.0282, gen loss = 1.0159
    Iteration 16350/39050: dis loss = 0.9922, gen loss = 1.8345
    Iteration 16400/39050: dis loss = 0.8945, gen loss = 2.1992
    Iteration 16450/39050: dis loss = 0.8297, gen loss = 1.2962
    Iteration 16500/39050: dis loss = 0.7001, gen loss = 1.8424
    Iteration 16550/39050: dis loss = 0.8681, gen loss = 0.9412
    Iteration 16600/39050: dis loss = 0.9564, gen loss = 2.0353
    Iteration 16650/39050: dis loss = 0.9824, gen loss = 0.9681
    Iteration 16700/39050: dis loss = 0.8706, gen loss = 1.0747
    Iteration 16750/39050: dis loss = 0.7600, gen loss = 2.1028
    Iteration 16800/39050: dis loss = 0.6218, gen loss = 2.0248
    Iteration 16850/39050: dis loss = 1.0256, gen loss = 0.7034
    Iteration 16900/39050: dis loss = 0.8090, gen loss = 1.1171
    Iteration 16950/39050: dis loss = 1.2065, gen loss = 0.6334
    Iteration 17000/39050: dis loss = 0.8034, gen loss = 1.1114
    Iteration 17050/39050: dis loss = 1.4048, gen loss = 0.4745
    Iteration 17100/39050: dis loss = 0.7936, gen loss = 1.8425
    Iteration 17150/39050: dis loss = 1.0153, gen loss = 0.7391
    
  • Iteration 17200/39050: dis loss = 0.8697, gen loss = 1.4361
    Iteration 17250/39050: dis loss = 0.9677, gen loss = 0.9314
    Iteration 17250/39050: dis loss = 0.9677, gen loss = 0.9314
    Iteration 17300/39050: dis loss = 1.0432, gen loss = 2.4892
    Iteration 17350/39050: dis loss = 1.0640, gen loss = 0.9013
    Iteration 17400/39050: dis loss = 0.6813, gen loss = 1.4769
    Iteration 17450/39050: dis loss = 0.7588, gen loss = 1.7723
    Iteration 17500/39050: dis loss = 0.7052, gen loss = 2.1375
    Iteration 17550/39050: dis loss = 0.8258, gen loss = 0.9004
    Iteration 17600/39050: dis loss = 0.8502, gen loss = 1.8127
    Iteration 17650/39050: dis loss = 0.6730, gen loss = 1.6425
    Iteration 17700/39050: dis loss = 0.9684, gen loss = 1.0495
    Iteration 17750/39050: dis loss = 1.1541, gen loss = 1.9938
    Iteration 17800/39050: dis loss = 1.0395, gen loss = 2.0989
    Iteration 17850/39050: dis loss = 1.1478, gen loss = 2.4993
    Iteration 17900/39050: dis loss = 1.1689, gen loss = 0.7209
    Iteration 17950/39050: dis loss = 0.6488, gen loss = 1.4780
    Iteration 18000/39050: dis loss = 1.0531, gen loss = 1.5826
    Iteration 18050/39050: dis loss = 1.0746, gen loss = 0.7425
    Iteration 18100/39050: dis loss = 0.7007, gen loss = 1.3145
    Iteration 18150/39050: dis loss = 0.8451, gen loss = 1.4895
    Iteration 18200/39050: dis loss = 1.2386, gen loss = 1.8009
    Iteration 18250/39050: dis loss = 0.6433, gen loss = 1.5910
    Iteration 18300/39050: dis loss = 0.4905, gen loss = 1.7876
    Iteration 18350/39050: dis loss = 0.8477, gen loss = 1.2819
    Iteration 18400/39050: dis loss = 0.8786, gen loss = 2.2671
    Iteration 18450/39050: dis loss = 1.0753, gen loss = 0.7873
    Iteration 18500/39050: dis loss = 0.7260, gen loss = 1.1887
    Iteration 18550/39050: dis loss = 0.8154, gen loss = 1.8391
    Iteration 18600/39050: dis loss = 0.7436, gen loss = 1.2242
    Iteration 18650/39050: dis loss = 0.8549, gen loss = 1.0022
    Iteration 18700/39050: dis loss = 0.9308, gen loss = 2.2570
    
  • Iteration 18750/39050: dis loss = 0.8463, gen loss = 1.7655
    Iteration 18800/39050: dis loss = 0.9384, gen loss = 2.4793
    Iteration 18800/39050: dis loss = 0.9384, gen loss = 2.4793
    Iteration 18850/39050: dis loss = 0.7623, gen loss = 1.1712
    Iteration 18900/39050: dis loss = 0.9517, gen loss = 1.7386
    Iteration 18950/39050: dis loss = 1.1101, gen loss = 1.9348
    Iteration 19000/39050: dis loss = 0.9056, gen loss = 1.9734
    Iteration 19050/39050: dis loss = 0.8016, gen loss = 2.4839
    Iteration 19100/39050: dis loss = 0.9806, gen loss = 1.0838
    Iteration 19150/39050: dis loss = 0.8661, gen loss = 1.7961
    Iteration 19200/39050: dis loss = 1.0674, gen loss = 0.7302
    Iteration 19250/39050: dis loss = 0.7350, gen loss = 1.2720
    Iteration 19300/39050: dis loss = 1.1568, gen loss = 2.6984
    Iteration 19350/39050: dis loss = 0.8209, gen loss = 0.9188
    Iteration 19400/39050: dis loss = 1.0516, gen loss = 0.7750
    Iteration 19450/39050: dis loss = 0.6501, gen loss = 1.8948
    Iteration 19500/39050: dis loss = 1.0316, gen loss = 0.6898
    Iteration 19550/39050: dis loss = 1.2874, gen loss = 0.6355
    Iteration 19600/39050: dis loss = 0.9836, gen loss = 1.9320
    Iteration 19650/39050: dis loss = 1.1005, gen loss = 0.9082
    Iteration 19700/39050: dis loss = 0.7644, gen loss = 1.7472
    Iteration 19750/39050: dis loss = 0.9334, gen loss = 1.1385
    Iteration 19800/39050: dis loss = 0.9947, gen loss = 0.9341
    Iteration 19850/39050: dis loss = 0.8457, gen loss = 0.7956
    Iteration 19900/39050: dis loss = 0.8259, gen loss = 1.1894
    Iteration 19950/39050: dis loss = 0.7384, gen loss = 2.0491
    Iteration 20000/39050: dis loss = 0.6983, gen loss = 1.0967
    Iteration 20050/39050: dis loss = 0.6698, gen loss = 1.9358
    Iteration 20100/39050: dis loss = 0.8765, gen loss = 1.3293
    Iteration 20150/39050: dis loss = 0.8228, gen loss = 1.1591
    Iteration 20200/39050: dis loss = 0.5705, gen loss = 2.1193
    Iteration 20250/39050: dis loss = 0.9508, gen loss = 1.2661
    Iteration 20300/39050: dis loss = 1.1404, gen loss = 2.2675
    
  • Iteration 20350/39050: dis loss = 0.7378, gen loss = 1.2225
    Iteration 20400/39050: dis loss = 1.3265, gen loss = 0.7893
    Iteration 20400/39050: dis loss = 1.3265, gen loss = 0.7893
    Iteration 20450/39050: dis loss = 0.8385, gen loss = 1.9051
    Iteration 20500/39050: dis loss = 0.8843, gen loss = 1.7837
    Iteration 20550/39050: dis loss = 0.9759, gen loss = 0.6087
    Iteration 20600/39050: dis loss = 0.5615, gen loss = 1.7242
    Iteration 20650/39050: dis loss = 0.6625, gen loss = 1.5110
    Iteration 20700/39050: dis loss = 1.0908, gen loss = 1.0533
    Iteration 20750/39050: dis loss = 0.9531, gen loss = 1.5758
    Iteration 20800/39050: dis loss = 1.1422, gen loss = 1.0520
    Iteration 20850/39050: dis loss = 0.6598, gen loss = 1.5584
    Iteration 20900/39050: dis loss = 0.7022, gen loss = 1.8766
    Iteration 20950/39050: dis loss = 0.7035, gen loss = 1.8396
    Iteration 21000/39050: dis loss = 0.7215, gen loss = 1.3437
    Iteration 21050/39050: dis loss = 1.0480, gen loss = 0.7264
    Iteration 21100/39050: dis loss = 1.2622, gen loss = 2.5912
    Iteration 21150/39050: dis loss = 0.8135, gen loss = 1.0429
    Iteration 21200/39050: dis loss = 0.6931, gen loss = 1.8413
    Iteration 21250/39050: dis loss = 0.8579, gen loss = 1.8581
    Iteration 21300/39050: dis loss = 0.9476, gen loss = 2.1245
    Iteration 21350/39050: dis loss = 1.2280, gen loss = 0.6498
    Iteration 21400/39050: dis loss = 0.6721, gen loss = 1.7755
    Iteration 21450/39050: dis loss = 0.7829, gen loss = 1.2415
    Iteration 21500/39050: dis loss = 0.7116, gen loss = 1.2469
    Iteration 21550/39050: dis loss = 0.9247, gen loss = 1.3638
    Iteration 21600/39050: dis loss = 0.9136, gen loss = 2.7215
    Iteration 21650/39050: dis loss = 1.1504, gen loss = 2.4780
    Iteration 21700/39050: dis loss = 0.8943, gen loss = 0.8575
    Iteration 21750/39050: dis loss = 0.7523, gen loss = 1.6259
    Iteration 21800/39050: dis loss = 0.8416, gen loss = 2.0712
    Iteration 21850/39050: dis loss = 0.8887, gen loss = 1.9761
    
  • Iteration 21900/39050: dis loss = 0.6672, gen loss = 1.9081
    Iteration 21950/39050: dis loss = 0.9083, gen loss = 0.8302
    Iteration 21950/39050: dis loss = 0.9083, gen loss = 0.8302
    Iteration 22000/39050: dis loss = 1.0922, gen loss = 2.2121
    Iteration 22050/39050: dis loss = 0.9250, gen loss = 2.0222
    Iteration 22100/39050: dis loss = 1.1649, gen loss = 0.7287
    Iteration 22150/39050: dis loss = 0.6595, gen loss = 1.7151
    Iteration 22200/39050: dis loss = 0.9767, gen loss = 0.8961
    Iteration 22250/39050: dis loss = 0.6795, gen loss = 2.1566
    Iteration 22300/39050: dis loss = 1.1168, gen loss = 0.8220
    Iteration 22350/39050: dis loss = 1.1141, gen loss = 2.0707
    Iteration 22400/39050: dis loss = 0.8961, gen loss = 1.5405
    Iteration 22450/39050: dis loss = 1.2012, gen loss = 0.6979
    Iteration 22500/39050: dis loss = 1.2519, gen loss = 0.6627
    Iteration 22550/39050: dis loss = 0.7262, gen loss = 1.4689
    Iteration 22600/39050: dis loss = 0.6504, gen loss = 1.3476
    Iteration 22650/39050: dis loss = 0.7799, gen loss = 2.5038
    Iteration 22700/39050: dis loss = 0.5802, gen loss = 2.4500
    Iteration 22750/39050: dis loss = 0.8849, gen loss = 2.1974
    Iteration 22800/39050: dis loss = 0.7732, gen loss = 1.7111
    Iteration 22850/39050: dis loss = 1.0473, gen loss = 2.6501
    Iteration 22900/39050: dis loss = 0.6286, gen loss = 1.4406
    Iteration 22950/39050: dis loss = 0.4874, gen loss = 2.1028
    Iteration 23000/39050: dis loss = 0.7453, gen loss = 1.9211
    Iteration 23050/39050: dis loss = 0.8720, gen loss = 0.9767
    Iteration 23100/39050: dis loss = 0.8444, gen loss = 1.2607
    Iteration 23150/39050: dis loss = 0.8449, gen loss = 1.2132
    Iteration 23200/39050: dis loss = 0.4722, gen loss = 1.6607
    Iteration 23250/39050: dis loss = 0.7943, gen loss = 1.2111
    Iteration 23300/39050: dis loss = 0.7778, gen loss = 2.7706
    Iteration 23350/39050: dis loss = 0.7828, gen loss = 1.1050
    Iteration 23400/39050: dis loss = 0.7001, gen loss = 1.1844
    
  • Iteration 23450/39050: dis loss = 0.6155, gen loss = 2.0324
    Iteration 23500/39050: dis loss = 0.6840, gen loss = 1.4397
    Iteration 23500/39050: dis loss = 0.6840, gen loss = 1.4397
    Iteration 23550/39050: dis loss = 0.7422, gen loss = 2.0256
    Iteration 23600/39050: dis loss = 0.6228, gen loss = 1.7990
    Iteration 23650/39050: dis loss = 0.8097, gen loss = 2.7257
    Iteration 23700/39050: dis loss = 1.2926, gen loss = 2.2068
    Iteration 23750/39050: dis loss = 0.9834, gen loss = 1.9410
    Iteration 23800/39050: dis loss = 0.6195, gen loss = 2.8101
    Iteration 23850/39050: dis loss = 0.6962, gen loss = 2.2349
    Iteration 23900/39050: dis loss = 0.7888, gen loss = 2.2223
    Iteration 23950/39050: dis loss = 0.6890, gen loss = 1.8531
    Iteration 24000/39050: dis loss = 0.5093, gen loss = 2.0333
    Iteration 24050/39050: dis loss = 0.7548, gen loss = 1.1450
    Iteration 24100/39050: dis loss = 0.6394, gen loss = 1.9979
    Iteration 24150/39050: dis loss = 0.8900, gen loss = 0.9650
    Iteration 24200/39050: dis loss = 0.6552, gen loss = 2.0563
    Iteration 24250/39050: dis loss = 0.5839, gen loss = 2.2146
    Iteration 24300/39050: dis loss = 0.7975, gen loss = 2.6882
    Iteration 24350/39050: dis loss = 0.7888, gen loss = 1.4659
    Iteration 24400/39050: dis loss = 0.8071, gen loss = 1.9530
    Iteration 24450/39050: dis loss = 0.7929, gen loss = 0.7984
    Iteration 24500/39050: dis loss = 0.9805, gen loss = 0.8416
    Iteration 24550/39050: dis loss = 0.8899, gen loss = 1.2353
    Iteration 24600/39050: dis loss = 0.8528, gen loss = 1.6514
    Iteration 24650/39050: dis loss = 0.6700, gen loss = 2.4146
    Iteration 24700/39050: dis loss = 1.4078, gen loss = 0.9007
    Iteration 24750/39050: dis loss = 0.9801, gen loss = 0.7418
    Iteration 24800/39050: dis loss = 0.8600, gen loss = 2.9252
    Iteration 24850/39050: dis loss = 0.6760, gen loss = 2.9656
    Iteration 24900/39050: dis loss = 0.9264, gen loss = 3.1154
    Iteration 24950/39050: dis loss = 0.8328, gen loss = 2.3529
    
  • Iteration 25000/39050: dis loss = 0.6798, gen loss = 2.6004
    Iteration 25050/39050: dis loss = 0.6452, gen loss = 1.4910
    Iteration 25050/39050: dis loss = 0.6452, gen loss = 1.4910
    Iteration 25100/39050: dis loss = 0.7914, gen loss = 1.7689
    Iteration 25150/39050: dis loss = 0.7299, gen loss = 1.3992
    Iteration 25200/39050: dis loss = 0.8895, gen loss = 2.4831
    Iteration 25250/39050: dis loss = 0.8259, gen loss = 1.8672
    Iteration 25300/39050: dis loss = 0.8876, gen loss = 2.4088
    Iteration 25350/39050: dis loss = 0.7983, gen loss = 2.6110
    Iteration 25400/39050: dis loss = 0.6153, gen loss = 1.4108
    Iteration 25450/39050: dis loss = 0.7043, gen loss = 2.0147
    Iteration 25500/39050: dis loss = 0.5950, gen loss = 1.5708
    Iteration 25550/39050: dis loss = 0.7549, gen loss = 2.6519
    Iteration 25600/39050: dis loss = 0.5030, gen loss = 2.1316
    Iteration 25650/39050: dis loss = 0.6716, gen loss = 1.7428
    Iteration 25700/39050: dis loss = 0.5792, gen loss = 1.9369
    Iteration 25750/39050: dis loss = 0.7483, gen loss = 1.1976
    Iteration 25800/39050: dis loss = 0.5596, gen loss = 1.9073
    Iteration 25850/39050: dis loss = 0.8715, gen loss = 3.4707
    Iteration 25900/39050: dis loss = 0.6101, gen loss = 1.7059
    Iteration 25950/39050: dis loss = 1.0593, gen loss = 3.2511
    Iteration 26000/39050: dis loss = 0.5930, gen loss = 1.5430
    Iteration 26050/39050: dis loss = 0.8316, gen loss = 1.1382
    Iteration 26100/39050: dis loss = 0.9597, gen loss = 0.6194
    Iteration 26150/39050: dis loss = 0.5299, gen loss = 1.7287
    Iteration 26200/39050: dis loss = 0.7883, gen loss = 1.1516
    Iteration 26250/39050: dis loss = 0.9198, gen loss = 2.0817
    Iteration 26300/39050: dis loss = 0.9637, gen loss = 1.0545
    Iteration 26350/39050: dis loss = 0.7158, gen loss = 1.4448
    Iteration 26400/39050: dis loss = 0.5265, gen loss = 1.7650
    Iteration 26450/39050: dis loss = 0.9319, gen loss = 2.9132
    Iteration 26500/39050: dis loss = 0.4912, gen loss = 2.0379
    Iteration 26550/39050: dis loss = 1.0144, gen loss = 2.6241
    
  • Iteration 26600/39050: dis loss = 0.8065, gen loss = 1.1375
    Iteration 26650/39050: dis loss = 0.9334, gen loss = 1.5080
    Iteration 26650/39050: dis loss = 0.9334, gen loss = 1.5080
    Iteration 26700/39050: dis loss = 0.5595, gen loss = 2.0797
    Iteration 26750/39050: dis loss = 0.6597, gen loss = 2.7287
    Iteration 26800/39050: dis loss = 1.1214, gen loss = 2.8068
    Iteration 26850/39050: dis loss = 1.0436, gen loss = 0.8414
    Iteration 26900/39050: dis loss = 0.4806, gen loss = 2.0615
    Iteration 26950/39050: dis loss = 0.5900, gen loss = 1.8554
    Iteration 27000/39050: dis loss = 0.7893, gen loss = 1.5012
    Iteration 27050/39050: dis loss = 1.0152, gen loss = 0.7925
    Iteration 27100/39050: dis loss = 1.1014, gen loss = 3.3496
    Iteration 27150/39050: dis loss = 0.7813, gen loss = 0.9494
    Iteration 27200/39050: dis loss = 0.9019, gen loss = 2.7015
    Iteration 27250/39050: dis loss = 0.6033, gen loss = 2.7551
    Iteration 27300/39050: dis loss = 0.3945, gen loss = 2.3656
    Iteration 27350/39050: dis loss = 0.8581, gen loss = 1.0374
    Iteration 27400/39050: dis loss = 0.5859, gen loss = 1.5850
    Iteration 27450/39050: dis loss = 0.7929, gen loss = 1.6085
    Iteration 27500/39050: dis loss = 0.4511, gen loss = 2.6602
    Iteration 27550/39050: dis loss = 0.6918, gen loss = 1.5759
    Iteration 27600/39050: dis loss = 0.4804, gen loss = 1.8751
    Iteration 27650/39050: dis loss = 0.4534, gen loss = 2.5051
    Iteration 27700/39050: dis loss = 0.6751, gen loss = 1.3118
    Iteration 27750/39050: dis loss = 0.7165, gen loss = 1.0777
    Iteration 27800/39050: dis loss = 1.0303, gen loss = 1.0308
    Iteration 27850/39050: dis loss = 0.5823, gen loss = 2.1668
    Iteration 27900/39050: dis loss = 0.9025, gen loss = 2.9388
    Iteration 27950/39050: dis loss = 1.1841, gen loss = 0.6804
    Iteration 28000/39050: dis loss = 0.6691, gen loss = 1.1224
    Iteration 28050/39050: dis loss = 0.7867, gen loss = 2.8204
    Iteration 28100/39050: dis loss = 0.4627, gen loss = 2.1472
    
  • Iteration 28150/39050: dis loss = 0.8183, gen loss = 2.6028
    Iteration 28200/39050: dis loss = 0.5853, gen loss = 3.1479
    Iteration 28200/39050: dis loss = 0.5853, gen loss = 3.1479
    Iteration 28250/39050: dis loss = 0.8611, gen loss = 0.6521
    Iteration 28300/39050: dis loss = 0.5103, gen loss = 2.2067
    Iteration 28350/39050: dis loss = 0.5758, gen loss = 1.9771
    Iteration 28400/39050: dis loss = 0.4588, gen loss = 2.8899
    Iteration 28450/39050: dis loss = 0.5404, gen loss = 1.8025
    Iteration 28500/39050: dis loss = 0.2268, gen loss = 2.7294
    Iteration 28550/39050: dis loss = 0.6758, gen loss = 3.4108
    Iteration 28600/39050: dis loss = 1.1962, gen loss = 0.5693
    Iteration 28650/39050: dis loss = 0.6337, gen loss = 3.7060
    Iteration 28700/39050: dis loss = 0.6816, gen loss = 1.3343
    Iteration 28750/39050: dis loss = 0.5841, gen loss = 1.8012
    Iteration 28800/39050: dis loss = 0.6899, gen loss = 3.1770
    Iteration 28850/39050: dis loss = 0.8264, gen loss = 1.1175
    Iteration 28900/39050: dis loss = 0.5087, gen loss = 2.5364
    Iteration 28950/39050: dis loss = 0.5353, gen loss = 2.3874
    Iteration 29000/39050: dis loss = 0.7318, gen loss = 1.0274
    Iteration 29050/39050: dis loss = 0.3842, gen loss = 2.1850
    Iteration 29100/39050: dis loss = 0.3840, gen loss = 2.1069
    Iteration 29150/39050: dis loss = 0.5222, gen loss = 1.5834
    Iteration 29200/39050: dis loss = 1.2000, gen loss = 4.0008
    Iteration 29250/39050: dis loss = 0.6258, gen loss = 2.1250
    Iteration 29300/39050: dis loss = 0.7491, gen loss = 3.1039
    Iteration 29350/39050: dis loss = 0.6214, gen loss = 2.2123
    Iteration 29400/39050: dis loss = 0.3977, gen loss = 2.3050
    Iteration 29450/39050: dis loss = 0.5469, gen loss = 2.9073
    Iteration 29500/39050: dis loss = 0.5876, gen loss = 2.5431
    Iteration 29550/39050: dis loss = 0.5233, gen loss = 1.7041
    Iteration 29600/39050: dis loss = 0.8627, gen loss = 2.2874
    Iteration 29650/39050: dis loss = 0.6243, gen loss = 2.8316
    
  • Iteration 29700/39050: dis loss = 1.1279, gen loss = 0.6590
    Iteration 29750/39050: dis loss = 0.6128, gen loss = 2.7713
    Iteration 29750/39050: dis loss = 0.6128, gen loss = 2.7713
    Iteration 29800/39050: dis loss = 0.4180, gen loss = 2.6501
    Iteration 29850/39050: dis loss = 1.2055, gen loss = 0.6685
    Iteration 29900/39050: dis loss = 0.8027, gen loss = 1.1676
    Iteration 29950/39050: dis loss = 0.9664, gen loss = 0.8570
    Iteration 30000/39050: dis loss = 0.6618, gen loss = 2.5774
    Iteration 30050/39050: dis loss = 1.1374, gen loss = 2.7770
    Iteration 30100/39050: dis loss = 0.4046, gen loss = 2.2823
    Iteration 30150/39050: dis loss = 0.6716, gen loss = 2.1577
    Iteration 30200/39050: dis loss = 0.4388, gen loss = 2.7895
    Iteration 30250/39050: dis loss = 0.4890, gen loss = 2.9045
    Iteration 30300/39050: dis loss = 0.9016, gen loss = 2.1186
    Iteration 30350/39050: dis loss = 0.6565, gen loss = 1.6448
    Iteration 30400/39050: dis loss = 0.6350, gen loss = 1.4655
    Iteration 30450/39050: dis loss = 0.4816, gen loss = 1.7241
    Iteration 30500/39050: dis loss = 0.7170, gen loss = 2.9416
    Iteration 30550/39050: dis loss = 0.5563, gen loss = 1.5663
    Iteration 30600/39050: dis loss = 0.3155, gen loss = 2.4888
    Iteration 30650/39050: dis loss = 0.7745, gen loss = 3.2175
    Iteration 30700/39050: dis loss = 0.7333, gen loss = 1.7256
    Iteration 30750/39050: dis loss = 1.2408, gen loss = 3.4982
    Iteration 30800/39050: dis loss = 0.6911, gen loss = 2.1077
    Iteration 30850/39050: dis loss = 0.6903, gen loss = 1.4123
    Iteration 30900/39050: dis loss = 0.7189, gen loss = 1.2900
    Iteration 30950/39050: dis loss = 0.9917, gen loss = 1.2928
    Iteration 31000/39050: dis loss = 0.6609, gen loss = 2.7723
    Iteration 31050/39050: dis loss = 0.9232, gen loss = 2.8852
    Iteration 31100/39050: dis loss = 0.6945, gen loss = 1.2475
    Iteration 31150/39050: dis loss = 0.4448, gen loss = 3.2756
    Iteration 31200/39050: dis loss = 0.4464, gen loss = 2.7041
    
  • Iteration 31250/39050: dis loss = 0.5570, gen loss = 1.9414
    Iteration 31300/39050: dis loss = 0.8045, gen loss = 3.3521
    Iteration 31300/39050: dis loss = 0.8045, gen loss = 3.3521
    Iteration 31350/39050: dis loss = 0.4723, gen loss = 2.0313
    Iteration 31400/39050: dis loss = 0.3730, gen loss = 2.5286
    Iteration 31450/39050: dis loss = 0.6191, gen loss = 1.4901
    Iteration 31500/39050: dis loss = 0.8337, gen loss = 2.9707
    Iteration 31550/39050: dis loss = 0.4774, gen loss = 3.2324
    Iteration 31600/39050: dis loss = 0.4837, gen loss = 1.8727
    Iteration 31650/39050: dis loss = 0.6328, gen loss = 3.3889
    Iteration 31700/39050: dis loss = 0.9088, gen loss = 3.6056
    Iteration 31750/39050: dis loss = 0.3642, gen loss = 2.2272
    Iteration 31800/39050: dis loss = 0.7317, gen loss = 1.2708
    Iteration 31850/39050: dis loss = 0.6024, gen loss = 2.6863
    Iteration 31900/39050: dis loss = 0.4863, gen loss = 2.6465
    Iteration 31950/39050: dis loss = 1.2884, gen loss = 3.5239
    Iteration 32000/39050: dis loss = 0.7387, gen loss = 1.4390
    Iteration 32050/39050: dis loss = 0.5672, gen loss = 1.9359
    Iteration 32100/39050: dis loss = 0.6425, gen loss = 2.9630
    Iteration 32150/39050: dis loss = 0.9962, gen loss = 0.9384
    Iteration 32200/39050: dis loss = 0.4693, gen loss = 1.8591
    Iteration 32250/39050: dis loss = 0.3203, gen loss = 2.5563
    Iteration 32300/39050: dis loss = 0.3104, gen loss = 3.0189
    Iteration 32350/39050: dis loss = 0.5383, gen loss = 1.5413
    Iteration 32400/39050: dis loss = 1.0619, gen loss = 2.6775
    Iteration 32450/39050: dis loss = 0.3993, gen loss = 1.8417
    Iteration 32500/39050: dis loss = 0.5359, gen loss = 3.4318
    Iteration 32550/39050: dis loss = 0.5900, gen loss = 1.4797
    Iteration 32600/39050: dis loss = 0.2625, gen loss = 2.8418
    Iteration 32650/39050: dis loss = 0.6003, gen loss = 2.1002
    Iteration 32700/39050: dis loss = 0.5850, gen loss = 2.7931
    Iteration 32750/39050: dis loss = 0.2801, gen loss = 2.2341
    Iteration 32800/39050: dis loss = 0.7013, gen loss = 1.6283
    
  • Iteration 32850/39050: dis loss = 0.5640, gen loss = 1.6046
    Iteration 32900/39050: dis loss = 0.4948, gen loss = 2.4163
    Iteration 32900/39050: dis loss = 0.4948, gen loss = 2.4163
    Iteration 32950/39050: dis loss = 0.5374, gen loss = 2.4138
    Iteration 33000/39050: dis loss = 0.5335, gen loss = 2.6799
    Iteration 33050/39050: dis loss = 0.6780, gen loss = 1.4455
    Iteration 33100/39050: dis loss = 0.7207, gen loss = 3.4161
    Iteration 33150/39050: dis loss = 0.5104, gen loss = 2.6655
    Iteration 33200/39050: dis loss = 0.5194, gen loss = 1.9709
    Iteration 33250/39050: dis loss = 0.4125, gen loss = 2.2557
    Iteration 33300/39050: dis loss = 0.5351, gen loss = 1.7529
    Iteration 33350/39050: dis loss = 0.5494, gen loss = 2.0983
    Iteration 33400/39050: dis loss = 0.5018, gen loss = 1.3082
    Iteration 33450/39050: dis loss = 0.3175, gen loss = 2.5561
    Iteration 33500/39050: dis loss = 0.5547, gen loss = 1.5941
    Iteration 33550/39050: dis loss = 0.6127, gen loss = 3.2081
    Iteration 33600/39050: dis loss = 0.3683, gen loss = 2.2387
    Iteration 33650/39050: dis loss = 0.6759, gen loss = 3.1007
    Iteration 33700/39050: dis loss = 0.3004, gen loss = 2.3446
    Iteration 33750/39050: dis loss = 0.8457, gen loss = 3.5467
    Iteration 33800/39050: dis loss = 0.2787, gen loss = 2.7753
    Iteration 33850/39050: dis loss = 0.5301, gen loss = 2.8378
    Iteration 33900/39050: dis loss = 0.7608, gen loss = 1.5738
    Iteration 33950/39050: dis loss = 0.6252, gen loss = 2.4615
    Iteration 34000/39050: dis loss = 0.9590, gen loss = 1.0053
    Iteration 34050/39050: dis loss = 0.5315, gen loss = 1.5393
    Iteration 34100/39050: dis loss = 0.7395, gen loss = 3.5992
    Iteration 34150/39050: dis loss = 0.6284, gen loss = 3.0709
    Iteration 34200/39050: dis loss = 0.6793, gen loss = 1.3831
    Iteration 34250/39050: dis loss = 1.1358, gen loss = 3.4535
    Iteration 34300/39050: dis loss = 0.6983, gen loss = 2.7475
    Iteration 34350/39050: dis loss = 0.7031, gen loss = 1.1707
    
  • Iteration 34400/39050: dis loss = 0.4541, gen loss = 2.0839
    Iteration 34450/39050: dis loss = 0.6386, gen loss = 3.1580
    Iteration 34450/39050: dis loss = 0.6386, gen loss = 3.1580
    Iteration 34500/39050: dis loss = 0.5920, gen loss = 1.7044
    Iteration 34550/39050: dis loss = 0.6148, gen loss = 2.0464
    Iteration 34600/39050: dis loss = 0.4319, gen loss = 2.5916
    Iteration 34650/39050: dis loss = 0.4702, gen loss = 2.4733
    Iteration 34700/39050: dis loss = 0.7118, gen loss = 3.4248
    Iteration 34750/39050: dis loss = 0.3060, gen loss = 2.9517
    Iteration 34800/39050: dis loss = 0.6234, gen loss = 2.4097
    Iteration 34850/39050: dis loss = 0.5108, gen loss = 1.8823
    Iteration 34900/39050: dis loss = 0.6235, gen loss = 1.7237
    Iteration 34950/39050: dis loss = 0.6602, gen loss = 1.9236
    Iteration 35000/39050: dis loss = 0.7509, gen loss = 3.2185
    Iteration 35050/39050: dis loss = 0.5487, gen loss = 1.7316
    Iteration 35100/39050: dis loss = 0.4190, gen loss = 3.4343
    Iteration 35150/39050: dis loss = 0.3938, gen loss = 2.5255
    Iteration 35200/39050: dis loss = 0.4949, gen loss = 3.4826
    Iteration 35250/39050: dis loss = 0.4513, gen loss = 1.9837
    Iteration 35300/39050: dis loss = 0.4204, gen loss = 2.4428
    Iteration 35350/39050: dis loss = 0.5629, gen loss = 1.6725
    Iteration 35400/39050: dis loss = 0.2029, gen loss = 3.0823
    Iteration 35450/39050: dis loss = 0.5090, gen loss = 3.4922
    Iteration 35500/39050: dis loss = 0.7392, gen loss = 2.1269
    Iteration 35550/39050: dis loss = 0.8170, gen loss = 3.8005
    Iteration 35600/39050: dis loss = 0.5254, gen loss = 2.0097
    Iteration 35650/39050: dis loss = 1.2199, gen loss = 3.2223
    Iteration 35700/39050: dis loss = 0.6812, gen loss = 3.1946
    Iteration 35750/39050: dis loss = 0.4573, gen loss = 2.3656
    Iteration 35800/39050: dis loss = 0.5613, gen loss = 2.2437
    Iteration 35850/39050: dis loss = 0.4729, gen loss = 2.2799
    Iteration 35900/39050: dis loss = 0.6893, gen loss = 3.8270
    
  • Iteration 35950/39050: dis loss = 0.4252, gen loss = 1.6196
    Iteration 36000/39050: dis loss = 0.5147, gen loss = 1.9309
    Iteration 36000/39050: dis loss = 0.5147, gen loss = 1.9309
    Iteration 36050/39050: dis loss = 0.4926, gen loss = 1.6812
    Iteration 36100/39050: dis loss = 0.4397, gen loss = 1.8400
    Iteration 36150/39050: dis loss = 0.6821, gen loss = 1.4659
    Iteration 36200/39050: dis loss = 0.5379, gen loss = 2.2689
    Iteration 36250/39050: dis loss = 0.5490, gen loss = 1.9351
    Iteration 36300/39050: dis loss = 1.6827, gen loss = 2.2989
    Iteration 36350/39050: dis loss = 0.3269, gen loss = 2.7260
    Iteration 36400/39050: dis loss = 0.6069, gen loss = 3.0031
    Iteration 36450/39050: dis loss = 0.4551, gen loss = 2.0690
    Iteration 36500/39050: dis loss = 0.7084, gen loss = 3.0779
    Iteration 36550/39050: dis loss = 0.9425, gen loss = 3.0884
    Iteration 36600/39050: dis loss = 0.4527, gen loss = 2.2251
    Iteration 36650/39050: dis loss = 0.8581, gen loss = 1.6284
    Iteration 36700/39050: dis loss = 0.3680, gen loss = 3.8401
    Iteration 36750/39050: dis loss = 0.8432, gen loss = 2.6192
    Iteration 36800/39050: dis loss = 0.5708, gen loss = 2.2373
    Iteration 36850/39050: dis loss = 0.5015, gen loss = 2.9714
    Iteration 36900/39050: dis loss = 0.7563, gen loss = 3.6967
    Iteration 36950/39050: dis loss = 0.8086, gen loss = 0.7774
    Iteration 37000/39050: dis loss = 0.6487, gen loss = 2.7577
    Iteration 37050/39050: dis loss = 0.7745, gen loss = 3.2842
    Iteration 37100/39050: dis loss = 0.6860, gen loss = 2.8333
    Iteration 37150/39050: dis loss = 0.8144, gen loss = 1.0187
    Iteration 37200/39050: dis loss = 0.3238, gen loss = 2.5127
    Iteration 37250/39050: dis loss = 1.0316, gen loss = 4.1832
    Iteration 37300/39050: dis loss = 0.4449, gen loss = 2.0489
    Iteration 37350/39050: dis loss = 0.6533, gen loss = 1.1449
    Iteration 37400/39050: dis loss = 0.5245, gen loss = 3.2078
    Iteration 37450/39050: dis loss = 0.8257, gen loss = 3.5838
    
  • Iteration 37500/39050: dis loss = 0.6765, gen loss = 1.2495
    Iteration 37550/39050: dis loss = 0.6253, gen loss = 3.2451
    Iteration 37550/39050: dis loss = 0.6253, gen loss = 3.2451
    Iteration 37600/39050: dis loss = 0.4285, gen loss = 2.7712
    Iteration 37650/39050: dis loss = 0.5156, gen loss = 3.3058
    Iteration 37700/39050: dis loss = 0.5574, gen loss = 1.3692
    Iteration 37750/39050: dis loss = 0.5148, gen loss = 2.2367
    Iteration 37800/39050: dis loss = 0.6086, gen loss = 1.5269
    Iteration 37850/39050: dis loss = 0.4603, gen loss = 3.0774
    Iteration 37900/39050: dis loss = 0.2877, gen loss = 3.1768
    Iteration 37950/39050: dis loss = 0.5093, gen loss = 4.1130
    Iteration 38000/39050: dis loss = 0.7242, gen loss = 3.5908
    Iteration 38050/39050: dis loss = 0.4614, gen loss = 2.0029
    Iteration 38100/39050: dis loss = 0.4997, gen loss = 2.3136
    Iteration 38150/39050: dis loss = 0.5010, gen loss = 2.0000
    Iteration 38200/39050: dis loss = 0.3209, gen loss = 2.6612
    Iteration 38250/39050: dis loss = 0.4119, gen loss = 3.4217
    Iteration 38300/39050: dis loss = 0.3702, gen loss = 2.5907
    Iteration 38350/39050: dis loss = 0.5749, gen loss = 2.5443
    Iteration 38400/39050: dis loss = 0.4553, gen loss = 1.6273
    Iteration 38450/39050: dis loss = 0.5416, gen loss = 3.5221
    Iteration 38500/39050: dis loss = 0.4416, gen loss = 2.3922
    Iteration 38550/39050: dis loss = 0.5691, gen loss = 3.2507
    Iteration 38600/39050: dis loss = 0.5878, gen loss = 1.2686
    Iteration 38650/39050: dis loss = 0.5750, gen loss = 1.8752
    Iteration 38700/39050: dis loss = 0.4241, gen loss = 2.1189
    Iteration 38750/39050: dis loss = 0.5801, gen loss = 1.6255
    Iteration 38800/39050: dis loss = 0.4598, gen loss = 3.1822
    Iteration 38850/39050: dis loss = 0.4482, gen loss = 2.8933
    Iteration 38900/39050: dis loss = 0.4477, gen loss = 1.8143
    Iteration 38950/39050: dis loss = 0.5049, gen loss = 2.7300
    Iteration 39000/39050: dis loss = 0.5075, gen loss = 3.1098
    Iteration 39050/39050: dis loss = 0.4882, gen loss = 1.8913
    
... Done!

Activation Maximization

GANs notoriously difficult to debug. Here we provide one of methods that are designed for investigating what networks learn, which can be helpful for debugging GANs.

Activation Maximization is a visualization technique to see what a particular neuron has learned, by finding the input that maximizes the activation of that neuron. Here we use methods similar to Synthesizing the preferred inputs for neurons in neural networks via deep generator networks.

In short, what we want to do is to find the samples that the discriminator considers most real, among all possible outputs of the generator, which is to say, we want to find the codes (i.e. a point in the input space of the generator) from which the generated images, if labelled as real, would minimize the classification loss of the discriminator:

$$\min_{z} L(D_{\theta}(G_{\phi}(z)), 1)$$

Compare this to the objective when we were training the generator:

$$\min_{\phi} \mathrm{E}_{z \sim q(z)}[L(D_{\theta}(G_{\phi}(z)), 1)]$$

The function to minimize is the same, with the difference being that when training the network we fix a set of input data and find the optimal model parameters, while in activation maximization we fix the model parameters and find the optimal input.

So, similar to the training, we use gradient descent to solve for the optimal input. Starting from a random code drawn from a standard normal distribution, we perform a fixed step of Adam optimization algorithm on the code.

The batch normalization layers should work in evaluation mode.

In [6]:
tf.reset_default_graph()
set_seed(241)

with tf.Session() as sess:
    with tf.device('/cpu:0'):
        dcgan = DCGAN()
        sess.run(tf.global_variables_initializer())
        dis_var_list = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, 'dis')
        gen_var_list = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, 'gen')
        saver = tf.train.Saver(dis_var_list + gen_var_list)
        saver.restore(sess, 'model/dcgan')
        actmax_results = dcgan.actmax(np.random.random([64, dcgan.code_size]))
        fig = plt.figure(figsize = (8, 8))   
        ax1 = plt.subplot(111)
        ax1.imshow(viz_grid(actmax_results, 1))
        plt.show()

The output should have less variety than those generated from random code. While it is reasonable that the samples that are "most real" makes up only a small portion of the sample space, this also gives us a hint that the so-called "mode collapse", in which the GAN simply fails to model a majority part of the data distribution, is a real problem.

A similar technique can be used to reconstruct a test sample, that is, to find the code that most closely approximates the test sample. To achieve this, we only need to change the loss function from discriminator's loss to the squared L2-distance between the generated image and the target image:

$$\min_{z} \left|\left|G_{\phi}(z)-x\right|\right|_2^2$$

This time, we always start from a zero vector.

In [25]:
tf.reset_default_graph()

with tf.Session() as sess:
    with tf.device('/cpu:0'):
        dcgan = DCGAN()
        sess.run(tf.global_variables_initializer())
        dis_var_list = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, 'dis')
        gen_var_list = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, 'gen')
        saver = tf.train.Saver(dis_var_list + gen_var_list)
        saver.restore(sess, 'model/dcgan')

        avg_loss, reconstructions = dcgan.reconstruct(test_samples[0:64])
        print('average reconstruction loss = {0:.4f}'.format(avg_loss))
        fig = plt.figure(figsize = (16, 16))   
        ax1 = plt.subplot(111)
        ax1.imshow(viz_grid(np.concatenate((test_samples[0:64], reconstructions), axis=2), 1))
        plt.show()
average reconstruction loss = 30.8770
{% include 'layout/components/footer.html' %} {% include 'layout/components/footer-scripts.html' %}